SYDE 676 Project Report – Fall 2002 Web Document Clustering Using Phrase-based Document Similarity

Measuring the similarity between documents is an essential operation in text mining, especially document clustering. The traditional method of finding the similarity between documents has always been based on extracting individual words from the documents, and using heuristics to give weights to those features. Standard methods in data mining are then used to find the similarity between documents using such features. In this project an information theoretic-based similarity measure is derived based on shared phrases between documents, rather than individual words. The basic concept is finding a metric that makes use of phrases rather than individual words. Two pairwise document similarity measures are proposed, one is corpus-dependent, and the other is corpus-independent. The corpus-independent measure allows for incremental processing of documents. Only the corpus-independent measure was evaluated in this report. The similarity measure is used for clustering web documents, which proved to have superior accuracy over traditional similarity measures. Evaluation of the clustering is performed based on Information Theory measures, specifically using the F-measure and Entropy.